Direct Search for Charged Higgs Bosons in Decays of Top Quarks 
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Abstract 



We present a search for charged Higgs bosons in decays of pair-produced 
top quarks in pp coUisions at ^/s = 1.8 TeV recorded by the D0 detector at 
the Fermilab Tevatron cohider. With no evidence for signal, we exclude most 
regions of the {Mjj± , tan /?) parameter space where the decay t — has a 
branching fraction > 0.36 and B{H^ tu-j-) is large. 
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The standard model (SM) relies on the Higgs mechanism for gauge-invariant generation 
of particle masses |l|]. It contains a single complex scalar doublet field, whose only observable 
particle is the neutral Higgs boson, H^. At present, no data limit the Higgs sector to a single 
doublet. In this Letter, we examine predictions of a two-Higgs-doublet model (THDM) that 
couples one doublet to up-type quarks and neutrinos, and the other to down-type quarks 
and charged leptons (Type-II model), just as in the minimal supersymmetric extension of 
the SM 0. For such coupling, flavor changing neutral currents are absent at tree-level 0. 
The additional degrees of freedom in this model provide a total of five observable Higgs 
fields: two neutral CP-even scalars and a neutral CP-odd scalar A^, and two charged 
scalars H^. In what follows, we report on a search for evidence of an extension of the Higgs 
sector, in the form of a boson, with the relevant parameters being its mass, Mh±, and 
the ratio of the vacuum expectation values of the doublets, tan p. 

In the SM, the primary decay of the t quark is t ^ W^h. The addition of the second 
Higgs doublet provides the t —* H^b mode, with a branching fraction B{t ^ H^h) oc 
{ml cot^ (3 + ml tan^ [3) + im^mf. This function has a minimum when tan P = ^Jmt/mb, and 
is symmetric in \ogiQ{taii (3) about this point. If tan/5 differs by about an order of magnitude 
from ^mt/mfj, the branching fraction becomes large, and decreases as Mh± increases. In 
this analysis, we assume B{t —>■ W^b) + B{t —>■ H^b) = 1. The masses of the three neutral 
scalars are assumed to be large enough to be suppressed in decays. Also, at tree level, 
there are no direct couplings to SM vector bosons. The only available decays of 
are therefore fermionic, with coupling proportional to fermion mass. For Mh± below ^110 
GeV, B{H+ T+u) ^ 0.96 for tan/3 > 2, and B{H+ ^ cs) ^ 1 for tan/3 < 0.4. Because 
of large coupling to the top quark 0, B{H^ t*b —>■ W^bb) becomes important and 

eventually dominant at higher values of Mjj± for tan (3 < ^mt/mb ■ 

D0 has carried out two searches for t H^b. An indirect search, which has been 
published looked for a decrease in the tt W~^W~bb signal expected from the SM, 
and the direct search, reported here, that searches for the — > t^u decay mode. Direct 
searches have been carried out by LEP experiments, and report a combined lower limit on 
Mh± of 78.6 GeV 0. CDF has also reported a direct search for H^, setting an upper limit 
on B{t —>■ H^b) in the range of 0.5 to 0.6 at 95% confidence level (CL) for masses in the 
range 60 to 160 GeV, assuming B{H^ — >• rVr) = 1 HI- 

In addition to the limits from the Tevatron and LEP, limits have also been published 
based on quantum corrections for Type-II THDM in other processes. CLEO sets a limit 
of Mh± > (244 -|- 63/(tan/?)^/^) GeV at the 95% CL from their inclusive measurement of 
6 ^ S7 [§. The L3 limit § on 5 -> r + z/^, leads Ref. to set a 90% CL limit of 
0.27 GeV^^ > (tan/?)/M^±. Finally, the branching ratios of r — > Ut-K and K i^ei'l), 
yields a limit of 0.21 GeV"^ > {tan (3) /Mh± at the 90% CL Although these limits 

exclude a larger part of available parameter space than our study, because of the difficulty 
of the measurements and ambiguities in theory, it is important to search for objects such as 
the in all possible channels, and not to defer entirely to theory. 

This analysis uses the same formulation and Monte Carlo (MC) tools as our indirect 
search. The theory is a leading-order perturbative calculation, requiring the t H^b 
coupling to be < 1, which limits the validity of our search to 0.3 < tan/? < 150. In addition, 
the calculation is unreliable for small \mt — Mh± \ and for large decay widths for t and H^. 
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This further hmits our search to Mh± < 160 GeV and B{t H+b) < 0.9. 

A direct search for divides naturally into two regions [jlT|: (1) small tan /?, where final 
states are dominated by jets, with imbalance in transverse momentum (Et), and (2) large 
tan (3, where the final state contains up to two r leptons and large missing transverse energy 
Because at small tan (3 there is background from multijet production, we concentrate 
on large tan j3 and ti —>■ rfz/T-z/T-+jets final states. The experimental signature for t —>■ H^h is 
nearly identical to that for t W^b. We therefore rely on the expected increase in absolute 
yield of r leptons at high tan (3 to differentiate between the two modes. 

The tt data for this analysis were obtained from pp collisions at y/s = 1.8 TeV ||12[| , 
and we consider both tt H^H^bb and tt —>■ H'^W^bb channels. Identification of the r 
relies on its hadronic decay modes, consisting primarily of one or three charged hadrons in 
a narrow jet, often accompanied by photons from tt^ decays, and a Ur- There are two b jets 
per event, and, when one of the top quarks decays to Wb, there are also two light quark 
jets (we consider only hadronic W modes). The event signature is therefore jets + with 
a roughly spherical distribution in the detector, and at least one narrow jet. Consequently, 
we rely on a multijet + trigger, which comprises 62.2 ±3.1 pb^^ of integrated luminosity 
(£). To reduce background, we use a set of loose selections, and then a neural network (NN) 
for more restrictive cuts. The loose criteria require that the event have > 25 GeV, at 
least 4 jets, each with Et > 20 GeV, but no more than 8 jets with Et > 8 GeV. 

We use a feed-forward NN based on jetnet [jl4|, with 3 input nodes, 7 hidden 
nodes, and 1 output node. The input variables are ^j., and two of the three eigenvalues 
of the normalized momentum tensor. The NN is trained on both signal (t H~^b), 
and background. The sample for training on signal, tt H~^H~bb, is generated using 
^ with both 



ISAJET 
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and H~ decaying to rz/^, and the r leptons to hadrons and u^. 
The response of the NN is relatively insensitive to the Mh±, we therefore use only one value, 
Mh± = 95 GeV. The same NN is also used for classifying tt H^W^bb channels, since the 



efficiency for this channel is comparable to that of the training sample. 

The primary sources of background are mismeasured multijet events, and W+ > 3 jet 
events. We therefore train the NN on a sample of 25,000 multijet events from data; even 
if the if^ exists, ~ 1 event is added to the sample. The W+ jets background is modeled 
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using VECBOS [|T6[ for parton production, and ISAJET for hadronization. Figure |l] shows 



the separation achieved for H"^ signal relative to our main background from multijet events. 
The chosen NN cutoff of 0.91, is based on a series of MC experiments used to determine 
the maximum sensitivity for if^. In the absence of signal, this also provides the maximum 
excluded area in {Mh±, tan (3) space. 

After applying the NN selection, we require that events have at least one hadronically 
decaying r lepton. The selection used in this analysis follows that of our W —>■ tVt 
study lll^. The principal requirement being the identification of one narrow jet in each 



event (ycr^ + o"| < 0.25, where the a correspond to the jet widths in r/, pseudorapidity, 
and 0, azimuthal angle), with 1 to 7 charged tracks, 10 < Et < 60 GeV for jets of cone 
R = \/ Af]'^ + = 0.5, and rejection of events with electrons or muons Ref. [|1^]. In 
addition to the criteria in Ref. [jl^, we require that the discriminant xl ~ xi > 0, where xl 
and Xb are the determined from a covariance matrix calculated from W tv^ MC, and a 
sample of multijet events respectively. The x^ fo^' the multijet sample uses the leading jet in 
each event {Et > 20 GeV). To define the covariance matrix, we use the fact that r-jets are 
narrower than normal hadronic jets in the energy range of our search. The variables used 
are the energy in each of the first five layers of our calorimeter, the log of the total energy, 
the ratio of the sum of the transverse energy of the two calorimeter towers with highest Et 
to the total jet energy, and ratios of jet energies in the central 3x3 and 5x5 calorimeter 
towers to the total jet energy. 

Because the measured values of au and rrit are based on the assumption that B(t 
W^h) = 1, it may be regarded as specious to use either in calculating the expected number 
of events. For ti production, we therefore use a QCD calculation giving ati = 5.5 pb |[T9|-pT|. 
Any possible contamination from ti — > H^W^bb, would affect the D0 measurement by 
< 5% for Mjj± < 140 GeV, we therefore use the value rrit = 175 GeV p2|p3| . The selection 
efficiencies for signal and background are listed in Table |. Using this information, we expect 
1.1±0.3 events from ti, 0.9±0.3 from W+ jets and 3.2±1.5 from mulitjet background, while 
we observe 3 events in the data. The jet energy, modeling of signal, and r identification, are 



the primary sources of systematic uncertainties. The first two are calculated as in Ref. [22 



while uncertainty in r identification is calculated as in Ref. [17|. 

Had if^ bosons been produced in ti decays, then the number of — > r + jets events 
would have exceeded expectation of the SM at high tan/3, because B{H^ t^i^t) = 0.96 



TABLE I. Cumulative efficiencies (in %) after the three stages of event selection for signal 
and background, for Mfj± = 95 GeV. The errors are statistical and systematic uncertainties added 
in quadrature. Event types are: (1) tt W^H^bb, W ^ qq , H ^ rur, (2) ti H^H^bb, 
H TUr, (3) ti W^W^bb, W tu^, W qq ; and (4) W+ > 3 jets, W ru-r, where we 



consider only r — > jet decays. 



Type 


Loose selection 


NN > 0.91 


r-id 


(1) 


50.0 ± 1.7 


18.3 ±0.9 


5.0 ± 1.0 


(2) 


35.2 ± 1.6 


12.9 ±0.9 


5.5 ± 1.0 


(3) 


45.1 ±2.0 


15.7 ± 1.0 


3.8 ±0.8 


(4) 


0.65 ± 0.04 


0.17 ±0.02 


0.04 ±0.01 
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in this region, while B{W~^ — rUr) = 0.11. Consequently, large tan/3 should be especially 
sensitive to contributions from H^. However, our data agree with the SM. Hence, to set a 
limit, we calculate the probability for data to fluctuate to the expectation from sources. 
Figure H shows the number of events observed, the number expected from SM processes, 
and the excess from for tan j3 = 150 and Mh± = 95 GeV, as a function of NN threshold. 
Above our NN cutoff of 0.91, there is clear inconsistency with the hypothesis of excess r 
production from sources. 

The probability that the number of expected events for a particular value of tan/5 and 
Mh± has fluctuated to the number of observed events (uobs), is given by the joint posterior 
probability density for Mh± and tan/5: 



P {Mh±, tan (3\nobs) oc J G{C) J G{nB) J G{A) x P{nobs\f^) dAdne dC, 



(1) 



where G represent Gaussian distributions, is the number of expected background events, 
and P{nobs\f^) is the Poisson probability of Uobs events, given expectation: /i(M/^-±, tan/5) = 
A{M jj±, tan P) a(tt) C+ub, where A{Mh±, tan [3) is the sum of the products of the branching 
fractions and efficiencies from all sources of ti decay. For a particular Mh±, and any tan (3, 
the value of A is computed via MC (in leading-order). The probabilities from Eq. |l] are 
then parameterized as a function of tan (3 for fixed values of Mh± , and fitted as a function 
of Mh± to obtain P {Mh±, tan i3\nobs), the Bayesian posterior probability density shown 
in Fig. ^. 

The prior probability distribution, as in the indirect search is assumed to be uniform 
over the allowed regions of Mh± and log(tan/?) and zero elsewhere. This gives equal weight 
to all possible branching ratios in Type II THDM. We further impose a lower limit on Mh± 
of 75 GeV, to provide an overlap with the limit from LEP experiments. The CL exclusion 
boundary in the {Mh±, tan/5) plane is obtained by integrating the probability density 
P {Mh±, tan (3\nobs) around a contour of constant P, such that the volume under the surface 
enclosed by that contour constitutes 95% of the volume under the full P{Mh±, tan P\nobs) 
surface. A 10% change in the ti cross section changes the excluded region by 10%, with the 
larger cross section yielding greater exclusion. The limits are shown in Fig. ^, along with 




NN Cutoff 

FIG. 2. Data and the number of events expected from all SM backgrounds (light), and from 
extra sources (dark) for tan (3 = 150 and M^± = 95 GeV, as a function of NN threshold. 
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FIG. 3. The normalized surface for P {Mjj±, tan P\nobs)- 
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FIG. 4. The region of exclusions at 95% CL in {Mfj±, tan /J) for mt = 175 GeV and 
a{tt) = 5.5 pb. (When statistical and systematic uncertainties become large, the Bayesian limit 
can depend on the distribution assumed for the prior probability.) 



results from our indirect D0 search, under the same assumptions. The exclusion region 
correspond to parameters that are < 5% likely. Because the indirect search excludes simulta- 
neously both large and small tan (3, the exclusion contour at high tan (3 represents approx- 
imately 2.5% of the volume under that posterior probability density surface. Also shown 
in Fig. ^ are the frequentist limits, wherein a point in the {Mh±, tan/3) plane is excluded 
when P {nobs\MH±, ta.n j3) < 5%, which is related to the posterior probability through Bayes 
theorem ||2^. Although the frequentist and Bayesian exclusion contours are shown on the 
same plot, they represent entirely different notions of probability ||24|| . 

In summary, our direct search for charged Higgs bosons in top quark decays shows no 
evidence of signal for Mfj± < 150 GeV. The region of small tan /3 does not provide r leptons 
through couplings to iJ^, and therefore cannot be excluded. At large tan/3, we extend the 
exclusion region beyond that of our indirect search. Assuming rrit = 175 GeV and a{tt) = 
5.5 pb, tan/3 > 32.0 is excluded at the 95% CL, for Mjj± = 75 GeV. The limits are less 



9 



stringent at larger Mh±, until Mh± = 150 GeV, where no limit can be set. Using the results 
of this Letter and those of our indirect search, we exclude B{t ^ H^b) > 0.36 at 95% CL 
in the region 0.3 < tan/3 < 150, and Mh± < 160 GeV. 
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